Surgical navigation system and method using audio feedback

ABSTRACT

A computer based system and method is disclosed for positional guidance in real-time surgical applications using audio feedback. The invention is based on translating the 5 spatial parameters of a surgical instrument or device, such as its position and velocity with respect to a given coordinate system, into a set of audio feedback parameters along the coordinates of a generalized audio space. Error signals which correspond to deviations of the actual instrument trajectory from an optimal trajectory stored in a computer memory are translated into a set of audio signals that indicate to the user whether correction is required. Accordingly, the system and method can be used in a variety of applications that require accurate special positioning. The audio feedback system and method of this invention employ a rich and comparatively unburdened sensory modality in the operating room and can be practiced independent from or along with standard visually-oriented systems and techniques used in medical pre-planning and/or virtual reality devices.

This application claims the benefit of U.S. Provisional Application No.60/035,945, filed Jan. 21, 1997.

FIELD OF THE INVENTION

This invention is directed to positional guidance systems and methodsand more specifically to a surgical navigation system and method usingauditory feedback.

BACKGROUND OF THE INVENTION

Modern pre-surgical modeling and planning systems are designed to assistsurgeons by allowing them to perform complex tasks such as planing andoptimizing tool-paths, manipulatings spaces, excising, harvesting,precisely locating transplantation sites, predicting postoperativeresults, and others. Such systems are designed to reduce the risks andunknowns in an operating room, and in this regard alone are capable ofsupplementing the surgeon's own skills and discretion considerably. Thefocus of much research in this area has been to develop interfaces thatcan efficiently, effectively, and ergonomically allow surgeons to accessvolumetric, functional, and trajectory-based navigational data frommodeling and planning sessions. The overwhelming thrust of research inthe area of surgical modeling and planning understandably has beenvisually oriented. Recent advances in medical imaging technology (CT,MRI, PET, etc.), coupled with advances in computer-based imageprocessing and modeling capabilities have given physicians anunprecedented ability to visualize anatomical structures in patients,and to use this information in diagnosis and treatment planning.However, visually based systems have proven to have performance problemswhen running in real-time, which typically renders them unusable exceptfor simulated surgery.

The use of virtual reality in computer-assisted surgical systems islimited in many ways by the present technological level. Thus, forexample, limited processing power and real-time rendering places tightconstraints on simulations in terms of the sophistication of visuallybased models. As models become more detailed, more resembling realobjects, greater processing power is needed. The head-mounted/heads-updisplay devices, which are ubiquitous in virtual reality systems areimpractical for surgical purposes because they interfere with thesurgeon's field of view, and their size and weight produces encumbranceand fatigue. Other factors such as the relatively low scan rates and lowresolution further limit the utility of head-mounted display technologyfor medical use. The devices are discomforting to wear for prolongedperiods of time and cause inevitable eyestrain. Furthermore, the latencyin image generation and in dynamic head tracking is noticeable in allhead-mounted display systems, but is most strikingly apparent inheads-up systems where synthetic imagery is overlaid over the real-worldvisage.

The implications for computer-assisted surgery based solely on visualprocessing are disappointing. It is simply unacceptable for thesurgeon's speed of motion during an operation to be limited by thedemands of the technology. The ultimate goal, after all, is for thetechnology to remove limitations rather than impose them. Poorlyconceived human-machine interface design adversely affects the abilityof surgeons to successfully perform procedures. This is unfortunatelythe case with many virtual reality systems which entangle the surgeonwith sensors and instrumentation. Ergonomic design must be a requirementof any system used in the operating room because many parameters alreadyinterfere with the intentions of the surgeon and the execution of thoseintentions by assisting devices. The limitations of the technologyshould not be further degraded by the interposition of poorly configuredinterfaces. Technology used in the operating room cannot only be usefulin itself, but must be intuitively useable in order to be functionallyuseful.

Another different but very important limitation of the commerciallyavailable technology is that the precision of image-based pre-surgicalplanning often greatly exceeds the precision of actual surgicalexecution. In particular, precise surgical execution has been limited toprocedures, such as brain biopsies, in which a suitable stereotacticframe is available. The inconvenience and restricted applicability ofsuch a frame or device has led many researchers to explore the use ofrobotic devices to augment a surgeon's ability to perform geometricallyprecise tasks planned from computed tomography (CT) or other image data.Clearly the ultimate goal of this research is a partnership between ahuman and machines (such as computers and robots), which seeks toexploit the capabilities of both, to do a task better than either can doalone. Clearly, computers can be very precise and can process largevolumes of data coming from any number of sensory feedback devices. Onthe other hand, a human surgeon is very dexterous, strong, fast, and ishighly trained to exploit a variety of tactile, visual, and other cues."Judgementally" controlled, the surgeon understands what is going on inthe surgery and uses his dexterity, senses, and experience to executethe procedure. However, in order to increase precision within acceptabletime limits or with sufficient speed, humans must be willing to rely onmachines to provide the precision.

U.S. Patents such as U.S. Pat. Nos. 5,546,943; 5,513,991; 5,445,566;5,402,801 and 4,905,163 discuss various devices which can be used toassist the surgeon's work. However, none of the prior art discusses in acoherent way the use of another information channel, the human auditorysystem, for the accurate processing by the surgeons of the huge volumeof information generated during an operation.

There exist a number of significant advantages to incorporating audiofeedback techniques into applications intended for real-time surgicaluse. The computational requirements for generating an audio signal aresubstantially smaller than for graphics, even though the auditorysensory modality is comparatively rich in bandwidth. Because auditoryperception is omnidirectional, it is possible to localize sounds emittedfrom any point in space, even from behind objects, whereas with visionit is only possible to localize objects falling within the viewingfrustum. Sound is capable of relating information about the relativedistance, azimuth, elevation, and the velocity of a sound source throughamplitude, spectral composition and Doppler shifting, respectively. Withadvanced techniques such as three-dimensional filtering, audio windowingand room acoustic simulation, it is also possible to relate theorientation of objects within a synthetic acoustic space. Theseobservations suggest that the area of information transmission in userinterfaces need not be constrained to the size of the monitor orhead-mounted display used. Information can emanate from anywhere inspace. The fact that audio feedback technology avoids many of theshortcomings of visual systems has already made it an attractive area ofexploration.

For example, it has been established that audio feedback can extend theavailable information transmission bandwidth significantly, in partbecause humans are capable of processing audio information in parallel.The vertigo caused by rendering latency and scanning, eyestrain, andvarious etiologies of simulator sickness almost universal in binocularthree dimensional systems are not an issue in audio systems.

Accordingly, it is perceived that audio-based real-time intraoperativesystems can considerably enhance the utility of modeling and planningtechnology to surgeons who cannot tolerate the encumbrance of graphicaldisplay hardware, and whose visual faculties have preexistingobligations. Presently available systems inadequately exploit theseadvantages of an independent or supplementary audio feedback system.Therefore, there is a need for a computer system and method for positionguidance using audio feedback providing spatial information.

SUMMARY OF THE INVENTION

The present invention concerns a novel computer method and system forpositioning guidance based on audio feedback.

In a preferred embodiment, the system of the present invention is basedon measurements of the spatial orientation and other positionalmeasurements. In particular, in accordance with the present inventionthe computer-based system for assisting a surgeon in positioning anarticle relative to a surgical target path in a patient comprises: meansfor determining a surgical target path based upon input patientinformation; sensor means for sensing surgical execution of a surgicaltarget path by the surgeon; and audio feedback means for real-timeadvising the surgeon based upon a comparison of the surgical target pathand the sensed surgical execution.

In a preferred embodiment, the system of the present invention furthercomprises a memory for storing one or more surgical target paths for thepatient. A surgical target path is expressed in a preferred embodimentin terms of two or more spatial coordinates, the values of which areindicative of the desired position and velocity of the article used inthe surgical procedure along the surgical target path. In anotheraspect, the system of the present invention further comprises a meansfor translating values of the two or more spatial coordinates obtainedfrom the measurement into corresponding values of two or morecoordinates of an audio space. In particular, in the system of thepresent invention each of the two or more coordinates of the audio spacemay correspond to an audio theme recognizable by the surgeon. Thus, forexample, this can be a consonant harmonic structure, such as a majortriad, each tone of which corresponds to values along a specific spatialcoordinate. In the present invention spacial coordinates are broadlyconsidered as positional (x-y-z) or angular coordinate, acceleration orothers.

In another aspect, the system of the present invention further comprisesmeans for automatically selecting different coordinates of the audiospace based upon the surgical target path and the sensed surgicalexecution, so that a surgeon can know how close his hand is moving to adesired path simply by listening to the audio feedback. Notably, unlikevisual systems, which require full attention from the human for certainperiods of time, an operating surgeon can correct his movementsvirtually without a distraction. Naturally, the system of the presentinvention can further be supplemented by corresponding visual feedbackmeans for advising the surgeon.

In accordance with another embodiment of the present invention, acomputer based positioning method is disclosed for assisting a surgeonin positioning an article relative to a surgical target path in apatient, comprising the steps of: determining a surgical target pathbased upon input patient information; sensing surgical execution of asurgical target path by the surgeon; and providing audio feedback forreal-time advising the surgeon based upon a comparison of the surgicaltarget path with the sensed surgical execution.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing aspects and other features of the invention are explainedin the following description, taken in connection with the accompanyingdrawings, wherein:

FIG. 1 shows a block diagram of the system hardware built in accordancewith a specific experimental embodiment of the present invention.

FIG. 2 shows a block diagram of the system hardware built in accordancewith a specific embodiment of the present invention.

FIG. 3 is a conceptual diagram depicting the relationship between thesoftware programs comprising the various subsystems and memory banks thesystem in accordance with the present invention.

FIG. 4 is a functional block diagram depicting the overall functioningand flow of data of the system used in a preferred embodiment of thepresent invention.

FIG. 5 shows a multidimensional error function used in a preferredembodiment of the present invention to advise the surgeon of deviationsfrom a target surgical path.

FIG. 6 shows a multidimensional error function employingthree-dimensional sound filtering of the signal used in a preferredembodiment of the present invention to advise the surgeon of deviationsfrom a target surgical path.

FIG. 7 illustrates an audio measurement (ruler) function used in apreferred embodiment of the present invention to assist the surgeon indetermining the distance between an origin and any desired point.

FIG. 8 shows a method for rendering into sound the geometry, density,histological type, or any other dimensions of the patient's anatomy orphysiology to be used in a preferred embodiment of the present inventionto advise the surgeon of deviations from a target surgical path.

FIG. 9 provides an illustration of the line of sight audio rulerfunction used in a specific embodiment.

FIGS. 10-11 provide an illustration of the granular synthesis-basedanatomical sonification function generated in a specific embodiment.

DETAILED DESCRIPTION

The foregoing aspects and other features of the invention are explainedin the following description, taken in connection with the accompanyingdrawings. There are a number of established methodologies for usingintegrated sound in computer applications and embedded systems. Soundhas commonly been employed as a redundancy measure in computer games andvirtual environment simulations by reinforcing other sensory elements inthe simulation, such as graphical or haptic. Symbolic sound structureshave also been used in the manner of icons in musical user interfaces(MUI's) developed for the blind (see Edwards, A.D.N., `Soundtrack: AnAuditory Interface for Blind Users,` Human Computer Interaction, 4(1),1989). With this approach musical sound, sound effects and sampledspeech form auditory icons [14], which may be manipulated within anaudio desktop space. See Wenzel, E. M.; Fisher, S.;

Stone, P. K., Foster, S. H. `A System for Three-Dimensional Acoustic"Visualization" in a Virtual Environment Workstation.` Proceedings ofVisualization '90 Conference. New York, N.Y.: IEEE Press. 1990.

Related approaches have included warning systems for civil aircraft andsimple audio feedback systems for medical equipment. See Patterson, R.D., `Guidelines for Auditory Warning Systems on Civil Aircraft.` (ReportNo. 82017). London, U. K.: Civil Aviation Authority, 1982; andPatterson, R. D., `Alarm Sounds for Medical Equipment in Intensive CareAreas and Operating Theaters,` Report No. AC598, Institute for Sound andVibration Research, University of Southampton, U.K.

Interface design for providing positional or orientational guidanceusing audio feedback is still in an experimental stage. Still, itincludes applications as diverse as an experimental three-dimensionalauditory `visualization` system, the NASA Ames Virtual InteractiveEnvironment Workstation (VIEW) system, and targeting for tacticalaircraft. See Mulder, J. D.; Dooijes, E. H. `Spatial Audio in GraphicalApplications.` Visualization in Scientific Computing. New York:Springer-Verlag; 1995. Wenzel, E. M.; Fisher, S.; Stone, 35 P. K.,Foster, S. H. `A System for Three-Dimensional Acoustic "Visualization"in a Virtual Environment Workstation.` Proceedings of Visualization '90Conference. New York, NY: IEEE Press. 1990; and U.S. Pat. No. 4,774,515.

These position guidance systems generally employ simple algorithms whichconvert a few dimensions of position data into corresponding acousticaldimensions. Additionally, three-dimensional audio processing filters arefrequently used to reinforce the sense of spatial position. E. M. Wenzeldescribes using frequency beat interference between two sound sources asa means for providing feedback for properly positioning a circuit boardin a NASA astronaut training simulations. This approach could easilyhave cross applications to surgical placement tasks.

The present invention is based, in part, on the understanding that theinterface design methodology employed by musical instruments may serveas a model for systems which aim to provide positional guidance by meansof audio feedback. Musicians control the pitch and various expressiveaspects of their performance by positioning their hands relative totheir instruments. In the case of variable pitchinstruments--instruments without discretized pitches--such as fretlessstringed instruments, the slide trombone, or the therein, the manualcontrol position must be extremely precise for the note to soundcorrectly. A variable spatial precision measurable in the extreme tofractions of a millimeter is a requirement for competent performanceupon an instrument such as the violin.

Instrument interfaces generally present a single axis of control withrespect to pitch, for instance the musician's finger sliding up and downa string. There are numerous other axes of control specific to eachinstrument used for controlling aspects of the performance, such asamplitude and spectral characteristics. In playing the classicelectronic musical instrument known as the therein, invented in 1928 byLeon Therein, the musician controls both pitch and amplitude by movinghis/her hands in the air, relative to two antennae.

Our invention is based on an conceptual inversion of the standardparadigm of performance upon a musical instrument, such as thetherein--using position to control sound--so that now sound can be usedto provide feedback as to position in surgical placement tasks.

Virtual Audio and Medicine

One of the specific advantages of incorporating audio feedback intomedical applications concerns the requirement for the surgeon tomaintain an extraordinary degree of focus on the task at hand. In thiscontext, information supplied by the assisting system must be intuitiveand information-rich without distracting the surgeon from the procedure.Integrated spatial audio systems function well in high-stressapplications. For example, in aircraft cockpits employing integratedaudio displays, the focus of attention between virtual sound sources andother information sources can be switched at will; whereas visionrequires physical movement. See Bly S. `Presenting Information InSound,` Proceedings of the CHI '82 Conference on Human Factors inComputing Systems. New York: ACM.

This has a clear cross-application to the demands of the surgeon.Conditions in the operating room are analogous in many ways to theaircraft cockpit, particularly in the audio dimension, where equipmentsounds emanate from many locations.

The System

A preferred system architecture for the present invention can begenerally described as two subsystems or groups; a presurgical modelingand planning system and a corresponding method, and the audio feedbacksurgical system substantially as described herein. The presurgicalsystem and method generally uses models of patient information to assistthe surgeon in planning precise surgical procedures. Any suitable typeof presurgical system and method could be used. For example, one can usethe system described in U.S. Pat. No. 5,445,166, which is incorporatedherein for all purposes.

Thus, for example, in one type of presurgical procedure, described belowfor informational purposes only, the principal components of thispresurgical procedure include a medical image database and displaysystem, an anatomical model builder, an anatomical feature extractor, asurgical simulator, an anatomical data base, and a surgical planoptimizer. These components are used to produce a surgical plan. Themedical image database and display system can be used, for example, tosupport archival, retrieval, low-level processing, and display of CT,MRI, and other images. The anatomical model builder transforms recordedimages into three-dimensional solid models of the patient's anatomy. Inthe second step (model reconstruction), a boundary representation ofeach connected set of tissue is constructed by an appropriate algorithm.In the third step, coplanar faces are merged to reduce the size of themodel somewhat. Implementation of this process can be semi-automatic, inwhich a technician "seeds" the search by identifying points on or nearridge curves, and the computer then locates and follows the ridgecurves. A more automatic procedure may, of course, be used.

The surgical system and method preferably includes a surgeon interfacesystem and passive manipulation aids. The surgeon interface uses avariety of modalities such as graphics, synthesized voice, tonal cues,programmable impedance of manipulator joints, etc., to provide online,realtime "advice" to the surgeon, based on the sensed relationshipbetween the surgical plan and surgical execution. A quite sophisticated,"intelligent" system can be used that uses its model of the surgicalplan to automatically customize displays, select appropriate sensortracking modes, and help interpret inputs from the surgeon. In thissystem, a helmet-mounted sterographic display could be used to projectthe surgical advice directly onto the surgeon's visual field, and thesurgeon could use voice input to tell the system what the surgeon wants.In a basic system, very simple real-time graphics and auditory cues canbe provided for alignment.

The second component of the system in accordance with the presentinvention is the audio feedback system. The challenge faced indeveloping a feasible methodology for designing a position feedbacksystem is to formulate an approach for transforming instrument andanatomical model data, of the type discussed above, into acomprehensible audio feedback signal that would be optimally intuitive,information-rich, ergonomic and economical in terms of the learningcurve for users.

Some notable obstacles to developing such a feasible methodology exist.Such are, for example, numerous psychoacoustic phenomena--especially thenonlinearity of human hearing--which conspires to perplex even simpleapplications, and the problem of determining frames of reference forusers. Basically, the problem of determining reference frames can bestated as the decision as to the appropriate mapping of axes of virtualspace to the user's coordinate system, and to the real world.

The exploitation of mapping reference frames in interface design is morecommon than one might imagine: consider how computer users have littletrouble overcoming the positional translations which occur between ahorizontally manipulated mouse, and a vertically projected cursor. Theinterface has a surprising intuitiveness despite the fact that the y andz axes may be reversed x offset +/-n depending upon the placement of themouse (and the handedness of the user) -possibly even the magnitude ofmovement scaled. The correspondence is simple and consistent; theexpected outcome of shifting the mouse in x, z does not diverge toosignificantly from the actual movement of the cursor.

Mapping geometry into sound is a larger strain upon the cognitivefaculties of the user than the simple transformations and scaling of thecomputer mouse to monitor/cursor interface. Instead of translating oneor more dimensions into another within a homomodal space, thesynesthetic approach involves transmodal mappings, in the case of thepresent invention the mapping of geometry into a corresponding auralspace. In order for the system to be at all useful, the cardinal task isto determine which dimensions of the initial modality are to be mappedto the resulting modality. For systems requiring as high a degree ofconfidence in performance as computer-surgical systems, the chosenconfiguration must result in the greatest possible intuitiveness of theinterface. Perceptual issues here become keenly important if thetransformation/mapping is desired to be as lossless as possible because,of course, the visual and auditory physiognomic and cognitive manifoldshave differing, even incomparable, perceptual resolutions (minimumperceptual differences), perceptual ranges, nonlinearities of response(e.g. frequency/intensity response in hearing), etc.

Musical Structural Functions

In addition to the basic audio primitives such as waveform, frequency,gain, phase, etc., musical informatics presents a large number ofcombinatorial structural functions which are open for consideration indeveloping a surgical sonification system. These allow significantflexibility in developing sonification methods. Some of these functionclasses are:

Harmonization: procedures for specifying simultaneous occurrences of n>1frequency sources (i.e., vertical procedures).

Counterpoint: procedures for specifying intervallic layering ofsimultaneously occurring melodic lines (i.e., horizontal procedures).

Rhythm: repetition patterns of temporal-proportional spacing of audioevents.

Modality: use of frequency-class sets (ordered pitch sets, consonance,dissonance).

Orchestration: juxtaposition of frequency and spectral relationstructures.

Large scale structure: global instances of repeated or related patternsin any musical dimension.

Development processes: strategic occurrence of the followingtransformations of large scale structures in any musical dimension(s):modulation, liquidation, augmentation, diminution, inversion,retrogression, etc.

Algorithmic procedures employing some of these high level functionscould impart a greater listenability and coherence to the feedbacksystem overall, as opposed to a purely low-level approach where thefeedback semiotic does not extend beyond the variation of physicalacoustical parameters, to which the ear might desensitized. They willalso allow the communication of multiple high-level coordinates in aartificial formal language to the surgeon.

Experimental Position Feedback System (System 1)

An experimental audio feedback system was assembled in accordance withthe principles of the present invention using inexpensive commerciallyavailable hardware. In a specific embodiment, the hardware platform waschosen more with an interest in rapidly assembling a working prototypewhich could be used to demonstrate the design concept than with theintent of assembling a prototype with a high quality of performance. Theprototype is based on the existence of a specialized real-timeaudio/synthesis programming kernel suited to perform the task on a lowlevel (MAX) for MacOS and uses a high-end Macintosh clone (FIG. 2, GroupB, Block 5) and programming the applications in MAX and C.

Hardware Description for System 1

In a preferred embodiment, the system host was a PowerPC 604e (225 Mhz)workstation running MacOS 7.5.5 (FIG. 2, Group B, Block 5). Thesonification subsystem consisted of a KORG 05R/W synthesis module (FIG.2, Group C, Block 7) with a 31.25 Kbit serial connection (FIG. 2, GroupC, Block 6) to the host computer. Stereo audio output is fed to a pairof headphones or an amplifier and speakers (FIG. 2, Group D, Block 8).The position tracking subsystem consists of a Polhemus 3Draw device(FIG. 2, Group A, Blocks 1-3) connected to the host via a CSI SCSIserial interface (FIG. 2, Group B, Block 4) running at 115,200 baud. Thesystem of the present invention in the specific embodiment discussedabove is further illustrated in FIG. 2.

Sound Synthesis and the MIDI Standard

The audio feedback system of the present invention was implemented in aspecific embodiment using commercially available synthesis modulesadhering to the MIDI standard. Naturally, pure software, or aDSP/software hybrid approach to synthesis can also be used inalternative embodiments. The audio devices of the selected system havemuch to offer in terms of reliability, usability, simplicity. The speedof operation and quality of sound is very high in comparison tosoftware-based systems because the hardware is dedicated solely towavetable playback. Sound programs can be designed, but they are more orless static during execution. Although the sounds can achieve a verysensuous quality with layering of many subprograms, the sounds havelimited application because all but a few parameters cannot becontrolled in real-time. This relatively restricted functionality is dueto the limitations of the Musical Instrument Digital Interface (MIDI)standard. The MIDI standard is a hardware specification and protocol forconnecting synthesis modules with computers. MIDI was designed forreal-time control of audio/synthesis devices. Transmissions under thisspecification operate at a rate of 31.25 Kbits/sec, asynchronous. Thecommunications signals, which consist of a start bit, an 8-bit data byteand a stop bit, contain control data, like switching a sound on and thenoff, changing the output pitch of an oscillator, or changing a presettimbre program, instead of transmitting actual sound waveforms. Thecontrolling computer produces a stream of amplitude and pitch envelopeinformation. See Roads, C., The Computer Music Tutorial, Cambridge: MITPress, 1996. The MIDI specification carries many limitations with it interms of the range of control. All control functions are limited to 128states. For instance, one cannot choose an arbitrary frequency of, say,440.09 Hz. The pitch control instruction is an integer within the range0-127. In fact, pitch instructions do not actually represent absolutepitches, but simply instruct the synthesis module to use the presetfrequency in storage that corresponds to that index, whatever that mightbe.

Implementation of Position Sonification: FIG. 6

As noted above, the sonification approach used in the system of thepresent invention was to some extent constrained by the MIDIspecification, which requires the use of feedback as a discrete ratherthan a continuous error function. The KORG synthesis module (FIG. 2,Group C, Block 7) allows a maximum of sixteen independent voices, oftenconsiderably fewer, depending upon the complexity of the sound program.In the following example, design applications for position feedback areonly discussed with reference to a single rigid body and neglectorientation and other dynamics (such as velocity and torsion). Theapplication employed a simple GUI for designing trajectories in threedimensional space, allowing a three dimensional resolution of 0.001 ofan inch (which is roughly equivalent to the Polhemus 3Draw device's(FIG. 2, Group A, Blocks 1-3) positional resolution of 0.005 of an inch). Two sound generating subsystems were employed as a part of the overallsystem: MIDI synthesis via an external synthesis module, and AIFF sampleplayback, using the Mac OS sound manager (FIG. 2, Group B, Block 5). TheMIDI subsystem (FIG. 2, Group C, Blocks 6-7) is used to generate adiscrete error function for error in the y, z planes, and the AIFFsubsystem to allow the user to tag specific locations on the target pathalong the x axis. A GUI allowed the user to specify the radii of theMIDI error envelopes, sound programs, pitch, and amplitude. A number ofAIFF samples of recorded speech and sound effects were provided. Theuser could set the sample volume, playback triggering radius from thetarget path center in the y, z plane, and the point along the targetpath in the x axis where playback would occur. As the instrument nearsits target path, the targeting feedback pitch for x approaches thelowest pitch of the consonant triad, thus, when the target is reached,harmonic stability is attained.

Preferred Embodiment Position Sonification System (System 2)

Hardware Description:

The Huron Digital Audio Convolution Workstation (FIG. 3, Groups B, C,Blocks 4-9) is a rack-mounting industrial PC (FIG. 3, Group B, Blocks4-6) fitted with a combination ISA bus and Huron bus backplane allowingstandard bus boards as well as Huron DSP boards (FIG. 3, Group C, Block8) and I/O boards (FIG. 3, Group C, Block 9) to be installed.

DSP Boards/Processors (FIG. 3, Group C, Block 8)

The Huron DSP board is a high performance, multiple-processor audio DSPengine. The Huron DSP board interfaces with the Huron bus, which is a256-channel 24-bit Time Division Multiplexed (TDM) audio bus, thatprovides low-latency communication to I/O boards and other Huron DSPboards in a Huron Digital Audio Convolution workstation. Up to 20 boardsmaybe installed in the Huron digital audio convolution workstationproviding significant audio signal processing capacity for audiorendering and other three-dimensional audio processing appropriate forthe project. The Huron architecture features 12 Mbytes of fast page-mode(zero wait state without page) Dynamic Random Access Memory (DRAM) and1.5 Mbytes of Static RAM (SRAM).

Each Huron DSP board supplies four processors of the Motorola D5P56002chipset (40 MHz clock). Each processor may read or write to any of the256 TDM channels which form the external Huron bus but may also read orwrite to an additional 512 TDM channels available on the DSP Board. Thisallows up to 768 TDM channels for inter-processor communications where256 channels allow communication to processors or audio interfaceslocated on other boards and 512 channels are for communication betweenprocessors on the board. The Huron DSP board supports up to 4 channelsof Lake Convolution (Lake's proprietary low-latency long convolutionalgorithm), allowing the implementation of FIR filters of up to 278,244taps in length without latency.

I/O Board (FIG. 3, Group C, Block 9)

The Huron I/O carrier board permits both analog and digital audiosignals to interface with the Huron system. The Huron I/O system is aflexible and modular design where compact, 2-channel interface modulesare installed on the Huron I/O carrier board as required for thevariable needs of our application. Up to eight I/O modules may beinstalled allowing 16 total I/O channels. Digital audio output isprovided via a 2-channel digital output module. Digital audio output isat the system sampling rate (48 kHz).

3-D Audio Filtering Tools (Implementation FIG. 8)

Our implementation employs Lake's sound field simulation andauralization software. The AniScape software system comprise thethree-dimensional spatialization component of the implementation of theinvention, in applications for the simulation of acoustic spaces forplayback over headphones or loudspeakers.

AniScape offers flexibility in the design of interactive virtualacoustic environments with moving sound sources, and control inreal-time. Simulations using this system provide a high degree ofrealism through the use of proven acoustic modeling methods, inconjunction with Lake DSP's long, low-latency convolution technology(Lake Convolution). AniScape gives total control of the locations ofmultiple sound sources and listeners within a virtual acoustic space.

Sound sources are encoded into a general three-dimensional audio formatby grouping their position and orientation, the position and orientationof the listener and the acoustic properties of the synthetic space theyinhabit. This generalized format is composed of just four standard audiosignals and may be decoded to headphones using a binaural decoder. Thesimulated acoustic space used is implemented in real-time using a highlydetailed room response.

Synthesis Approach

Following is a discussion of a number of synthesis techniques which areapplicable to the problem of developing DSP-based audio synthesisalgorithms, but not inclusive of all possible or preferred techniques.In this context, this discussion is only intended to suggest certainsound synthesis methods with respect to the described example feedbackalgorithms.

Wave-table Lookup: Fundamentals of Synthesis

The basis of at least one class of synthesis approaches is a wave-tablelookup-based oscillator implementation. In our example hardwareconfiguration this is implemented by using banks of sine-waveoscillators on the Motorola DSP56002 chipset. With the availability ofprecision digital signal processors, such as the Motorola DSP56K family,stable and low distortion sine waves of arbitrary frequency can beproduced using wave table look-up with interpolation to reducedistortion. The wave-table is scanned by means of an index that isincremented at each sample period. This table-lookup approach isindustry-standard, and serves as the basis for granular synthesis aswell as additive synthesis.

Granular Synthesis: Clouds of Sound (FIGS. 10 to 11)

Granular synthesis has a useful application in sonification because itpermits the sonification of gradients using variable density clouds ofsound particles. In a granular system, sound can be viewed in terms ofboth wavelike properties and particulate properties just as light energy(photons). Granular synthesis aggregates acoustic events from thousandsof sound grains. These sound grains typically last from 1 to 100 ms.This range of duratmin approaches the minimum perceivable time for eventduration, amplitude, and frequency discrimination. A particularanatomical object three-dimensional volume may be propagated with aparticular type and density of sound grain. Penetration of this objectwould result in a unique type of sound being generated.

Granularity proves to be a useful model for understanding complex soundphenomena. Complex waveforms can be viewed as constellations ofprimitive units of energy, where each unit is bounded in time andfrequency space. There are numerous parallels between granular synthesisand wavelet theory.

The grain proves to be a useful representation because it combinestime-domain information (envelope and waveform) with frequency-domaininformation (waveform period inside the grain, waveform spectrum). Thisis different from representations that do not capture frequency-domaininformation, and Fourier-derived representations that presume thatsounds are summations of infinitely long sinusoids.

An amplitude envelope shapes each grain. This envelope can vary inimplementation from a Gaussian curve to nonlinear functions. Complicatedenvelopes, like band-limited pulses, describe resonant grains that soundlike woodblock taps in sparse textures when the grain duration fallsbelow 100 ms. Narrow envelopes create popping textures when the totalgrain duration falls to less than 20 ms. Sharp changes in the envelopefunction cause strong perturbation of the spectrum. This effect is dueto the convolution of the envelope's spectrum with that of the grainwaveform. The grain duration can be constant, random, or it can vary ina frequency-dependent way. This implies that we should assign shorterdurations to high-frequency grains. The waveform within the grain, inthe case of our system, is synthetic, and is the sum of sinusoidsscanned at a specified frequency.

Several parameters are varied on a grain-by-grain basis: duration,envelope, frequency, spatial location, waveform (a wave-table). Thisgrain-by-grain level of control leads to some unique spectral effectsthat are only possible by using this method.

In a preferred embodiment, granular synthesis is implemented usingeither a simple sine wave oscillator (as described above) controlled byan envelope generator or a wave-terrain approach. In opposition to thesimplicity of the sine-wave oscillator, the generation of even a simplesound requires a massive quantity of control: thousands of parametersper. These parameters describe each grain in terms of starting time,amplitude, etc. Since it is cumbersome to specify the parameters of eachgrain programmatically, a higher-level system of organization isrequired. This system should generate the grain specifications.

The complex spectrum generated by granular synthesis is proportional tothe quantity of control data. If n is the number of parameters for eachgrain, and d is the average grain density per second of sound, it takesd * n parameter values to specify one second. Since d typically variesfrom a few dozen and several thousand, it is clear that for the purposesof compositional control, a higher-level unit of organization is needed.The purpose of such a unit is to allow the programmer to createinstances of large quantities of grains. The synthesis method embodiedby the algorithm listed we use can be classified using this granularorganization model. We describe this model as an asynchronous cloudmodel. Clouds of a specific density are formed relative to the densityof the anatomic structure or tissue being sonified. These clouds aremapped to three-dimensional space using a spatialization algorithm.Cloud microstructure is generated stochastically. Refer to FIGS. 10 and11 for conceptual depictions of this method.

Wave Terrain Synthesis: Extracting Waveforms from Anatomy

This synthesis technique proceeds from the fundamental principle ofwave-table lookup as discussed above. It is possible to extend the basicprinciple of wave-table lookup to the scanning of n-dimensional wavesurfaces or volumes.

A traditional wave-table can be visualized in two dimensions as afunction wave(x) indexed by x. A two-index wave terrain can be plottedas a function wave(x, y) on a three-dimensional surface (e.g. thesurface of an anatomical object model). The z-point represents awaveform value for a given pair (x, y). The waveform is stored in atable so defined and is a function of two variables. A scan over theterrain is an orbit. Although the astronomical term "orbit" connotes anelliptical function, the orbit can consist of any sequence of points onthe wave terrain.

Any three-dimensional surface can serve as a wave terrain, from aconstrained function to an arbitrary projection. As in techniques likefrequency modulation and wave-shaping, the advantage of using simplefunctions is the predictability of the derived waveform and spectrumgenerated by a given wave terrain function. The following conditionsmust be met in order to predict the derived waveform:

Both the x and y functions and their first-order partial derivatives arecontinuous (in the mathematical sense) over the terrain. Both functionsx and y are zero on the terrain boundaries. The second property ensuresthat the functions and their derivatives are continuous when the orbitskips from one edge of the wave terrain to another edge. Such a skip isanalogous to the right-to-left wraparound that occurs in one-indexwave-table scanning. A terrain which satisfies these conditions isdefined by the following equation:

    wave(x,y)=(x-y)*(x-1)*(x+1)*(y-1)*(y+1)

The signal generated by wave terrain synthesis depends on both the waveterrain and the trajectory of the orbit. The orbit can be a straight orcurved line across the surface, a random walk, a sinusoidal function, oran elliptical function generated by sinusoidal terms in both the x and ydimensions.

When the orbit is fixed, the resulting sound is a fixed waveformcharacterized by a static spectrum. A way to generate time-varying (andconsequently interesting, or memorable) waveforms is to change the orbitover time. Our implementation of this approach is an extension where theorbit is fixed but the wave terrain is variable over time. In this case,the wave-scanning process employs three-dimensional surfaces found inthe anatomical dataset. The position of the instrument inthree-dimensional space, which is used as a cursor for the surgeon, alsofunctions as an index for the terrain scanning process. The resultingwaveform is time-discretized to form grains.

The wave terrain technique provides a robust model in the context of asonification method intended for providing feedback with respect to thenavigation of a cursor through and over three-dimensional volumes andsurfaces. In this approach the three-dimensional surface of any objectmay be interpreted as a wave terrain. Upon intersection or penetrationof an object by the surgical instrument, the algorithm may orbit at afixed or programatically defined frequency and scan path across a regionof the surface, defined by a projective function perpendicular to somepredetermined axis of the instrument, and the surface normal of somevoxel closest to the point of contact. This is, in effect, the sonicanalogue of a surface rubbing.

Three-dimensional Audio Spatialization Filterinq

While human beings are generally considered sight-dependent creatures,there is no disputing the importance of auditory cues in our ability torelate to the environment. At the physiologic level, sound waves aretransmitted to the inner ear via a system of mechanical interactionbetween membranes, small bones, and channels containing a fluid medium.In the inner ear, the sound waves of particular frequencies aredeflected in such a way as to disturb the position of hair cells thattrigger neuronal connections traveling through the auditory nerve to thecerebral cortex. These impulses are interpreted by the brain as soundsof a particular pitch and intensity.

The sense of hearing includes the ability to locate sound sources inthree-dimensional space: "It was Lord Rayleigh (1907) who proposed thatwe localized sound sources by exploiting intensity and phase differencesbetween the signals from the left and right ears." 57 Moreover, theimpact of each individual's head-shape and external ear on the reflectedsound waves received by the inner ear is crucial for sound localization.

Research by Shaw (1974) demonstrated that the pinna has a significantinfluence on shaping the spectral envelope of incident sound.Furthermore, this spectral shaping is dependent upon the spatial originof the sound source. Thus the brain learns to extract spatialinformation from the unique `earprint` the pinnae impress upon theincoming pressure waves." Each individual therefore receives the soundwaves generated by an auditory source in a slightly different way, andthen, using cues based on phase and intensity differences and theinformation derived from the impact of one's pinnae and head on thesound waves, can localize the sound source in three dimensions,including azimuth, elevation, and distance from the listener.

More specific investigation of what factors influence sound localizationhas added four other parameters in addition to the factors of interauraltime delay, head shadow, pinna response, and shoulder echoes thatcomprise the "Head-Related Transfer Function." They include head motion,vision, intensity, and early echo response and reverberation caused bylocal acoustics

The particular interference characteristics of an individual'shead-shape and pinnae on the transfer of sound waves to the ear canalsis a measurable function that has generated one approach to virtualsound modeling. Various techniques involving speaker arrays andsensitive miniature microphones inserted into the ear canal make itpossible to derive an individual's "Head-Related Transfer Functions(HRTFs)," which actually include the impact of the head, shoulders, andexternal ear on the incoming sound waves. Once the characteristics ofparticular sound waves that the listener localizes to a specific pointis known, these sound waves can potentially be reproduced artificially,in order to give the listener the impression that the sound source islocated in a specific place, whatever the location of the speakersgenerating the actual sound waves.

However, incorporating sound into a virtual reality application can beaccomplished in a number of vastly different ways, with widely differentintentions. At the most fundamental level, immersive virtual realityapplications are given increased validity and realism when they make useof natural-seeming audio effects, even when such effects are not thatclosely tied to the visual environment, as they exploit the naturalcognitive tendency of the listener to associate logically-associatedsensory inputs: "Although a loudspeaker may be displaced from the actuallocation of a visual image on a television or movie screen, we caneasily imagine the sound as coming from an actor's mouth or from apassing car. This is an example of what is termed visual capture; thelocation of the visual image `captures` the location derived from audiocues." These effects can be as simple as the triggering of an unmodifiedpre-stored audio sound when the user acts in a particular way in thevirtual environment. They need not be highly sophisticated aural effectscalculated for each particular user in order to have a significanteffect on the quality of the virtual environment: "A New York Timesinterviewer, writing on a simulation of a waterfall. . . described how`the blurry white sheet` that was meant to simulate a waterfall througha $30,000 helmet-mounted display seemed more real and convincing withthe addition of the spatialized sound of the water." 61 In thisparticular case, the addition of a comparatively cheap and easy toincorporate technology, generally appreciable by any user, considerablyimproved the overall impression of the simulation.

The vast potential of aural feedback to improve the quality of virtualreality systems is clearly at present largely underutilized, and, in thefield of medical applications, virtually untested: "high-resolutioncolor graphic hardware and software have been around longer on personalcomputers than the audio equivalent, CD-quality two-channel digitalsound." This potential, however, should be obvious even to the observerunfamiliar with the state of virtual reality technology. Simply stated,"one might be able to work more effectively within a VR environment ifactions were accompanied by appropriate sounds that seemingly emit fromtheir proper locations, in the same way that texture mapping is used toimprove the quality of shaded images." At the most basic level,therefore, pursuing the potential applications of audio technology forvirtual reality is a fruitful avenue of research.

The benefits of the more sophisticated types of audio feedback invirtual reality are potentially much greater: "Although digitized soundsamples play an important role in these systems, it is the ability toshape a waveform and adjust features such as pitch, timbre, amplitude,phase and decay that make it an important technology for VR. While aspreviously noted, digital imaging technology, volume-rendering, andvisual display technology are currently stretching the limitations ofcurrently-available computer processing speed and memory, in the case ofaudio technology, "current generation hardware is already capable ofsupplying binaural signals that model the attenuation of pressure wavesentering the user's ear canals, and thus simulate the way our earsinfluence perceived sounds in the real world."

Sonification Algorithms

Listed below are several algorithms used in accordance with a preferredembodiment of the present invention.

    ______________________________________                                        Algorithm 1: Discrete Error Function                                          ______________________________________                                         1  loop                                                                       2    serial.sub.-- write( polhemus, request.sub.-- datapoint                  3    update.position = serial.sub.-- read( polhemus                           4    midi.sub.-- plan = which.sub.-- plan( midi, update.position.x            5    if( update.position.yz == midi˜lan.position.yz                     6      serial write( midi, midi.sub.-- plan.note.sub.-- on                    7    else                                                                     8      serial write( midi, midi.sub.-- plan.note.sub.-- off )                 9    endif                                                                   10      x.sub.-- function.note on = which.sub.-- note( update.position.x,             midi.sub.-- plan, trajectory.sub.-- len.x )                           11      serial write( midi, x function.note on                                12      aiff.sub.-- plan = which.sub.-- plan( aiff, update.position.x )       13      if( update.position.yz == aiff.sub.-- plan.position.yz )              14        play.sub.-- aiff ( aiff.sub.-- plan.aiff, aiff.sub.-- plan.volum              e )                                                                 15      endif                                                                 16  endloop                                                                   ______________________________________                                    

FIG. 6 provides an illustration of the discrete error function in the y,z plane, using the above algorithm and the hardware system of FIG. 2.

    ______________________________________                                        Algorithm 2: Beat Interference                                                ______________________________________                                         1  ref.sub.-- frequency = a                                                   2  loop                                                                       3  cursor = get.sub.-- instrument.sub.-- position()                           4  plan = find nearest.sub.-- point( target.sub.-- path, cursor )             5  if( cursor != plan )                                                       6      oscillator.sub.-- a << ref.sub.-- frequency                            7      oscillator.sub.-- b << ref.sub.-- frequency - ( plan - cursor          8    else                                                                     9      oscillator.sub.-- a << ref.sub.-- frequency                           10      oscillator.sub.-- b << ref.sub.-- frequency                           11    endif                                                                   12  endloop                                                                   ______________________________________                                    

FIG. 7 provides an illustration of the beat interference error functionas generated using the algorithm set forth above, and implemented uponthe preferred embodiment hardware system depicted in FIG. 3. Anextension of this algorithm, employing three reference frequencies andsix oscillators, is also depicted.

    ______________________________________                                        Algorithm 3                                                                   ______________________________________                                        1   loop                                                                      2     cursor = get.sub.-- instrument.sub.-- position()                        3     plan = find.sub.-- nearest.sub.-- point( target.sub.-- path, cursor           )                                                                       4     spatial.sub.-- location = plan - cursor                                 5     dsp << convolve( error.sub.-- signal, spatial location )                6   endloop                                                                   ______________________________________                                    

FIG. 8 provides an illustration of the three-dimensional error functionas generated using the algorithm set forth above, and implemented uponthe preferred embodiment hardware system depicted in FIG. 3. Thisthree-dimensional filtration algorithm may also be employed as aredundancy measure and extension to any of the other algorithms byproviding three-dimensional spatialization of their output audiosignals.

    ______________________________________                                        Algorithm 4                                                                   ______________________________________                                         1  loop                                                                       2   loop                                                                      3    if( select.sub.-- point() )                                              4     origin = get.sub.-- instrument.sub.-- position()                        5     exit loop                                                               6    endif                                                                    7   endloop                                                                   8   set tracking.sub.-- device.sub.-- origin( origin )                        9   loop                                                                     10    cursor = get.sub.-- instrunent position()                               11    radius = cartesian.sub.-- distance( cursor, origin )                    12    if( 0 = modulo(radius, large.sub.-- increment )                         13     oscillator << large.sub.-- increment.sub.-- click                      14    elsif( 0 = modulo( radius, small.sub.-- increment )                     15     oscillator << small.sub.-- increment.sub.-- click                      16    endif                                                                   17   endloop                                                                  18  endloop                                                                   ______________________________________                                    

FIG. 9 provides an illustration of the line of sight audio rulerfunction as generated using the algorithm set forth above, andimplemented upon the preferred embodiment hardware system depicted inFIG. 3.

    ______________________________________                                        Algorithm 5                                                                   ______________________________________                                        loop                                                                           2   cursor = get.sub.-- instrument.sub.-- position ()                         3   if( intersected.sub.-- object( cursor )                                   4    object = get.sub.-- object( cursor )                                     5    if( random() <= object.density )                                         6     wave.grain.sub.-- function = make.sub.-- grain( object.tissue.sub.-        - type )                                                                   7     wave.amplitude = make.sub.-- amplitude( object.density,                cursor.velocity )                                                              8     ... // Map other parameters                                             9     oscillator << wave                                                     10    endif                                                                   11   endif                                                                    12  endloop                                                                   ______________________________________                                    

FIGS. 10-11 provide an illustration of the granular synthesis-basedanatomical sonification function as generated using the algorithm setforth above, and implemented upon the preferred embodiment hardwaresystem depicted in FIG. 3.

It should be understood that the foregoing description is onlyillustrative of the invention. Various alternatives and modificationscan be devised by those skilled in the art without departing from thespirit of the invention. Accordingly, the present invention is intendedto embrace all such alternatives, modifications and variances which fallwithin the scope of the appended claims.

We claim:
 1. A computer based system for positioning an article relativeto a surgical target path using audio feedback, comprising:a memorystoring a surgical target path expressed in terms of two or more spatialcoordinates along a three dimensional surface representing a model of ananatomical object, the values of said spatial coordinates beingindicative of a desired trajectory of said article along the surgicaltarget path; one or more sensors tracking the actual trajectory of thearticle during surgical execution; a comparator providing on outputdifferences between the desired trajectory and the actual trajectory ofthe article along said two or more spatial coordinates; a processortranslating the provided differences along said two or more spatialcoordinates into corresponding two or more coordinates of an audiospace; and audio means for playback of said two or more coordinates ofthe audio space to assist in positioning the article relative to thesurgical target path.
 2. The system of claim 1 wherein said processortranslates the provided differences into sound grains of predeterminedduration.
 3. The system of claim 2 wherein the duration of the soundgrains is between 1 and 100 ms.
 4. The system of claim 2 whereinparameters characterizing the sound grain are in the group comprising:duration, envelope and frequency.
 5. The system of claim 1 wherein theprocessor translates the provided differences based on user-specificinput.
 6. The system of claim 1 wherein the memory stores two or moresurgical target paths associated with a patient.
 7. The system of claim1 wherein the memory stores two or more surgical target paths associatedwith different patients.
 8. The system of claim 1 wherein at least oneof said two or more coordinates of the audio space corresponds to arecognizable audio theme.
 9. The system of claim 1 wherein the processorcomprises means for automatically selecting different coordinates of theaudio space based upon the surgical target path and the actualtrajectory of the article during surgical execution.
 10. The system ofclaim 1 further comprising visual means for advising the surgeon. 11.The system of claim 1 further comprising correction means for changingthe surgical target path during surgery and determining a new surgicaltarget path based, at least in part, upon previously sensed surgicalexecution.
 12. The system of claim 11 wherein the correction meanscomprises a voice responsive input means.
 13. The system of claim 1further comprising means for advising the surgeon, said means foradvising including means for automatically providing a resistance forceto motion of the article in at least one degree-of-freedom.
 14. Thesystem of claim 1 wherein the article is a surgical instrument.